Models of Consensus for Validation of Expert Systems

Models ofconsensus are used in the validation process to develop a basis for system performance. This paper develops two analytic models ofconsensus that can be useful in the validation process. The first model employs the binomial model to study the probability that the consensus judgment is correct or incorrect. This problem is critical for the validation of expert systems, since the model leads us to conclude that in some cases consensus judgment is not appropriate. That basic model is extended to account for both different levels of expertise and unequal prior odds. The second model is a Bayesian model of the use of consensus judgment and the validation process. There are two applications of the Bayesian model The first application ofthe Bayesian model finds that, in some cases, consensus judgment is not appropriate and to generate a hypothesis as to the conditions for the use of the consensus judgment. The second application compares an expert systems performance to the consensus judgment. It can be used to assist in deriving stopping criteria for the: validation process.